Dusting off the Server and Making a Blog
ORA Suspiciously Easy Migration to NixOSIt can't be that hard, right? I mean, I already have a virtual server to host it on, set up with Nginx and an FTP server (among other things). I could just point a subdomain to a folder, handwrite some HTML files, and be done with it!
Though there is one problem I want to solve: I have a bad memory, and that makes running a server difficult. I mean, it's probably been two years since I've done any maintenance on it, and it's been running without issue, but I honestly don't even know what is running on it.
So the problem to solve: Make it easy to manage this server despite only checking on it once every few months.
Figuring out What I Even Have
I'm going to try and keep this part short since it will mostly be about specific software I use and mistakes that I have made. If you don't want to read about bad approaches, just skip ahead to where I'm Doing Something About It.
Part 1: Nginx
Anyway, where to begin. The first thing I want to check is Nginx.
I know I'm running at least two web servers, and I use Nginx as a reverse proxy for those.
And how do I check that?
Well, what I do is open my terminal, hit ^R
, and search vi
until I find a result that looks like it was editing a Nginx config file
(and eventually I found this: sudo vi /etc/nginx/conf.d/nodesite.conf
).
Searching through previous commands is currently one of my main ways of getting things done,
and I despise it.
For every important command I've run, there are ten more I ran just seeing what would work,
and all those bad commands pollute my history.
Anyway, it appears I have five different server configs in the file named nodesite.conf
.
None of which point to web servers that run node by the way.
Two of them actually point to web servers I shut down long ago,
and one still works, but I completely forgot it existed.
I'm beginning to gather a list of problems I want solved. So far I have
- Clean up my Nginx config.
- Have a better way of getting to the config (no more
^R
searching in bash)
But I run more than just web servers.
Part 2: systemd
Services
Any good sysadmin would have their services neatly packed in systemd
units
(right...? I spend little time running servers...).
I, unfortunately, looked in /etc/systemd/system/
and I don't see anything written by
myself.
I can tell that I have MariaDB installed, but not much more than that.
Since this really wasn't getting me anywhere,
I decided on opening htop
out of desperation
and found something really quite horrifying...
This
For a more clear explanation:
This is part of what htop
shows when I use it to see which processes are running.
As we can see, I have tmux
open,
running multiple instances of bash
where I can then run my two services and view the logs.
While this does allow me to have these things running when I disconnect,
all of this is lost when the server restarts, and in general is a horrible way to run services.
is how I run my services:
None
htop
Snippit
A portion of my htop
output.
That's... got to change. I swear, it was long ago that I set this all up — I'm not like this anymore.
Looking through the rest of htop
I also see I have Postgres and Fail2Ban running,
and apparently one of those web servers I previously stated I shut down is here running as well.
This is too much, even for a hobby server. This is so unorganized and poorly done that my draft of this blog post has become my most thorough documentation of this server so far. What I have is entirely unmanageable — so unmanageable that I didn't even notice Pure-FTPD running until days later (only adding this part during a revision). A server where I can lose my services is not a server I want to run.
So to add to my list:
- Use actual
systemd
services when it makes sense - Settle on one database
- Have all services that I run be easily discoverable
Doing Something About It
Of course my plan never was to actually clean this up. Why fix the old server when I can make a whole new one without any problems in the first place! Of course, I will need to make sure these old problems don't arise again, but I have a plan for that.
The OS
And the plan? NixOS! Discussions on NixOS online can be fun — there's a lot that people love, and a lot that people hate — and after quite a bit of reading, I've decided that switching to NixOS is in fact my best course of action.
But what is there to love, and what is there to hate? Or first, what even is Nix?
Well Nix is a few things. There is Nix the programming language, Nixpkgs the package manager and package collection, and NixOS the Linux distribution. Together (or apart, but with less success), these tools can be used for declarative, reproducible system builds.
I once had a server that needed a certain set of software installed and configured. I wrote a tutorial on how to get it all installed in case someone ever messed things up and needed to start from scratch. To do this, I set up a virtual machine, and tried to install everything necessary. Every time I did a step correctly, I'd save a snapshot. Every time I made a mistake, I'd revert. In the end, I had a server perfectly set up with everything it needed, and nothing else. Not a single file out of place. I could have made an image and put it directly on the real hardware if I wanted, or just follow the steps.
Using NixOS is like doing this,
but for a computer to read instead of a human.
I find it funny how writing it this way makes it sound like I've
discovered writing software after only ever using a REPL before.
Like You mean I can type all my code into a
You edit a config file, writing a tutorial of sorts of how to set up the server, then when you run
.py
file and run it again and again and it always works the same?
I don't need to exit out, start over, and re-type anything if I make a big mistake?
That's so cool!sudo nixos-rebuild switch
, Nix follows the tutorial and sets up your server.
Everything perfectly done, software, services, configuration, all at once.
Not a single file out of place — like magic.
Though maybe you messed up though and it did leave files out of place.
Instead of booting up your last saved snapshot and trying again,
just undo your last changes to the config file and try again.
So, there's the "what" and the reason to love, but why the hate?
Well, you see, I've never used an operating system more difficult than NixOS. A few months ago, I needed to set someone up with their own server and website and decided to use Nix. It took me 4 days getting a new virtual server up and running with just Nginx and Spring Boot. Normally that would have taken me... 20 minutes? Or if I hit the "have Ubuntu pre-installed" checkbox, 15? But no, I did it the Nix way. I had to struggle to get the OS installed in the first place, learn a whole new language just to configure the server, debug strange error messages, copy odd configs I found online, and more. But in the end, I got exactly what I needed, nothing more, nothing less. It was perfect.
But that was months ago, and while I learned a lot, I've since forgot most of it. So now I need to do it all again.
Time to actually get started.
Installing NixOS on Hetzner
Unfortunately, Hetzner doesn't offer a NixOS image when setting up a server,
nor does it allow me to supply my own image.
Luckily, Nix has a resource
specifically for this!
Since I don't remember which way I installed it last time,
I'll be following the nixos-infect
approach.
This approach allows me to start with an Ubuntu server (which Hetzner allows me to easily make)
and switch it to a NixOS server.
After purchasing another virtual server (I can't let my services shut down, there would be at least one person that that would disappoint!) and fiddling with the login credentials, I have a fresh Ubuntu server. Time to infect it with Nix.
Infection is as simple as running the following command, though it does kind of scare me.
Bash
Location: The new server
Make sure to set NIX_CHANNEL
to the most recent version!
curl https://raw.githubusercontent.com/elitak/nixos-infect/master/nixos-infect | NIX_CHANNEL=nixos-23.05 bash -x
Annnnd... that's it? It looks like it worked. I remember this being difficult.
Configuring NixOS
Maybe this is when It's supposed to get difficult.
The first thing I went to check out was my config in /etc/nixos/configuration.nix
and,
well,
I don't have NeoVim yet.
Anyway, this is what it looked like fresh out of the box.
none My terminal after opening the file
# GNU nano 7.2 /etc/nixos/configuration.nix
{ ... }: {
imports = [
./hardware-configuration.nix
./networking.nix # generated at runtime by nixos-infect
];
boot.tmp.cleanOnBoot = true;
zramSwap.enable = true;
networking.hostName = "dvs-hnix";
networking.domain = "";
services.openssh.enable = true;
users.users.root.openssh.authorizedKeys.keys = [''ssh-rsa AAAAB3NzaC1yc2EAAA >
system.stateVersion = "23.11";
}
#^G Help ^O Write Out ^W Where Is ^K Cut ^T Execute ^C Location
#^X Exit ^R Read File ^\ Replace ^U Paste ^J Justify ^/ Go To Line
I've got two things I need to do. One, install NeoVim so I can comfortably edit my configuration, and two, make myself a user account so I'm not doing this all as root. Conveniently, I can do those both in this configuration file.
For now, this is all I need to add to my configuration.
nix
/etc/nixos/configuration.nix
The newly added lines...
users.users = { # This defines the users that the system will have.
main = { # I always name my user `main`. It's just a preference.
isNormalUser = true; # This is a normal user, not a system user.
extraGroups = [ "wheel" "main" ]; # wheel is for sudo, main for myself.
packages = with pkgs; [ # This is how I add packages for just myself.
git # Ever want to use version control for a whole server? Use git to
# manage your configuration.nix, its basically the same thing.
htop # Just useful to have.
tree broot # Tree is self explanatory, broot is like an interactive tree.
tmux # Again, just useful to have.
ccze goaccess # These are for viewing access.log files. I quite like them.
];
openssh.authorizedKeys.keys = [ # Just add your ssh keys here. Pretty easy.
''ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCinTQBCdbJTs2HnPnceib3czD21t >
];
};
};
# Creates the group called `main`.
# I like to have a group with the same name as my user.
users.groups.main = {};
# These packages are available to everyone
environment.systemPackages = with pkgs; [
neovim # I didn't forget about vim!
];
After saving and closing, I run the magic command sudo nixos-rebuild switch
.
After adding some missing semicolons and adding an argument to the beginning closure
(my full working config will be at the end of this section), my build succeeded.
I'm now able to log out of root, and log back in as main.
As mentioned in the comments of the config above,
versioning the /etc/nixos/
directory with git makes NixOS all the more magical.
Separation of Concerns Configs
While I won't ssh into the server often, I do want a somewhat comfortable user config.
The way I'm hoping to organize my configuration is to have the main
configuration.nix
file describe only what the server has and does.
That includes software installed, services (and their configs) that are running,
and the user accounts available.
What that doesn't include though, are the user specific configs and extra software.
To be more specific: my server config includes the main
user because
that user is important for managing the server.
My main
user has
git
, htop
, tmux
, etc., installed
because I believe those are important programs to have while managing
this
server.
My main config will also (eventually) hold my Nginx configs and my system services because
those are things that my server does.
What my main config will not have however, is any configuration for Git or NeoVim because those
configurations are important to me, not to my server.
Why this distinction? Having a separation between what my server does and what I like makes it easier both for me to work on servers that don't belong to me, and people that aren't me to manage servers that are primarily mine. How will it make it easier? Because all the relevant information and only the relevant information will be grouped together. If someone wants to work on the server, everything they need and nothing they don't need will be in one place and vice versa.
If you are enjoying the simplicity and pain free nature of NixOS or have no interest in running Home Manager, please skip the next section and go straight to Bringing my Services Back
Home Manager
I'm beginning to get to the point where I'm less and less familiar with everything that I'm doing. I know that a thing called Home Manager is what I want Is it though? Given my explanation in the prior section, Home Manager seemed to be exactly what I wanted, and it has lived up to that expectation. Future me (who has finished the migration and is now revising my writing) is less sure that this was the right move to make. I've made a conscious decision to avoid flakes this whole time because they just didn't feel right, and Home manager is beginning to feel this way to me as well. I do plan on sticking with it until I have a solid reason not to though, and when I get around to setting up my desktop, I do plan to install it there too. , but I don't know how to install it. If you're wondering why I'd bring that up right after linking to a page that has an entire section on installing Home Manager, let me first remind you of why I wanted NixOS in the first place. I had said that when working in NixOS, everything would be in the config files, and the config files would be all that is needed to build the server up from scratch. I'm absolutely terrified of global mutable states, so running commands outside the config that are important to the system setup is something I'm just unwilling to do. Imagine my surprise then, when the only two installation methods for Home Manager start with the same step...
To make the NixOS module available for use you must import it into your system configuration. This is most conveniently done by adding a Home Manager channel to the root user. For example, if you are following Nixpkgs master or an unstable channel, you can run
bash
sudo nix-channel --add https://github.com/nix-community/home-manager/archive/master.tar.gz home-manager sudo nix-channel --update
So what are these Nix channels, why do they exist if they are so un-nix like, and how do I avoid them?
Despite Nix having a page on just this,
I still had to dig around to get an answer that satisfied me.
It seems that Nix channels are places to get software that directly involve the host OS.
The official channels (and the ones that the Nix documentation talks about)
are all self-explanatory in how they relate to the OS... because they are the OS.
There are channels like nixos-unstable which is the unstable branch of NixOS,
nixos-24.11
,
the latest stable version of NixOS, nixpkgs-24.11-darwin
,
the apple version of the Nix package manager, and a few others along similar lines.
The ones that aren't in this official list though, are from the
Nix Community Projects repo.
These also seem to be directly related to the OS, but in a different way.
There is NixOS-WSL
which I can only assume is important for running NixOS in a WSL
environment,
home-manager
which directly integrates into the Nix build process itself,
and 195 others than I assume somehow directly integrate into NixOS too.
So it seems that these have to exist in a different way than how regular packages are installed, but I'm still unhappy with the way home manager recommends installing itself. I can understand using commands instead of a config for very important system updates, like upgrading NixOS to the next major version, or switching to the unstable branch, but not for the addons. Luckily, I did find a way to add Home Manager using the configs, though I did have to learn about the nix language first.
Previously, lines 2
through 6
of my
/etc/nixos/configuration.nix
look like this:
nix
/etc/nixos/configuration.nix
(previous version) Lines 2-6imports = [ ./hardware-configuration.nix ./networking.nix # generated at runtime by nixos-infect ];
But I then changed them to this:
nix
/etc/nixos/configuration.nix
Lines 2-8
imports = let
home-manager = builtins.fetchTarball "https://github.com/nix-community/home-manager/archive/release-24.11.tar.gz";
in [
./hardware-configuration.nix
./networking.nix # generated at runtime by nixos-infect
(import "${home-manager}/nixos")
];
So what is going on here?
Before, imports
was directly assigned a value, and that value was a list of strings
(that happened to be relative file paths).
Now, I'm doing almost the same thing, but with a non-local file.
The let ... in ... ;
syntax was a weird one for me.
The let
defines zero or more variables,
then the in
allows a single expression to follow where those variable definitions are valid.
That single expression can have a {...}
which itself would have more expressions inside,
or in my case have a list [...]
.
Crucially, this whole block evaluates to the value of the second ...
,
in my case, the list.
What this gives me is the ability to make a temporary variable named home-manager
,
fill it with an archive found at the given URL,
then import a file from that archive and add it to the list.
That list is then returned and assigned to be the value of imports
.
I believe it may be possible to manage the current version of NixOS similarly to this,
but that's a task for another time.
Now I can get on with the configuring.
Configuring With Home Manager
As outlined in the Separation of Configs section,
I want my Home Manager config to live outside the main /etc/nixos/configuration.nix
file.
Normally, if not in the main config, it would live in the user's home directory,
but I decided I actually want it in the /etc/nixos/
directory anyway.
My reasoning is that I, the sysadmin, need to control the server,
and happen to do that by ssh
-ing into my main
account.
Viewing it this way, the Home Manager config is the server's config for me
and not my config for the server.
If I were setting up NixOS as my desktop OS (which I plan to do in the future),
I would see it the other way around and place the Home Manager config in my user directory.
With that out of the way I can start to work on the config. After some searching online and playing around, I decided all I want for now is a good NeoVim setup, and I ended up with the following:
nix
/etc/nixos/extra-user-config.nix
(New File)
Created by me. Name is arbitrary.
{ pkgs, ... }: {
home-manager.users.main = { # Configuring for my `main` user
home.stateVersion = "23.11" # This is the version of NixOS FIRST installed.
programs = {
neovim = {
enable = true;
viAlias = true;
# Find more plugins using `nix-env -f '<nixpkgs>' -qaP -A vimPlugins`
plugins = [
# Visual Editor Additions
pkgs.vimPlugins.vim-fugitive
pkgs.vimPlugins.vim-gitgutter
pkgs.vimPlugins.nerdtree
# Language Related
pkgs.vimPlugins.nvim-treesitter
# Styling
pkgs.vimPlugins.sonokai
];
# And an escape, where we can append lines directly to the config.
extraConfig = ''
set number relativenumber
set ignorecase
set expandtab
set shiftwidth=4
set tabstop=4
color sonokai
autocmd FileType nix setlocal shiftwidth=2 softtabstop=2 expandtab
'';
};
};
};
}
My main configuration.nix
file needs to be updated to reference this now.
This can be achieved simply by just adding a line to the imports:
The first 9 lines of /etc/nixos/configuration.nix
. Note line 7, the new
addition.
nix
/etc/nixos/configuration.nix
Lines 1-9; Read line 7.
{ pkgs, ... }: {
imports = let
home-manager = builtins.fetchTarball "https://github.com/nix-community/home-manager/archive/release-24.11.tar.gz";
in [
./hardware-configuration.nix
./networking.nix # generated at runtime by nixos-infect
./extra-user-config.nix # My user configuration ### NEW ADDITION ###
(import "${home-manager}/nixos")
];
Interestingly, the first line of my new extra-user-config.nix
file looks just like that of the
configuration.nix
, why is that?
It's actually a function
definition.
The { pkgs, ... }:
portion defines the arguments
Technically, this isn't fully correct, but it is really quite close.
What is show here is actually a function definition with a single parameter
that happens to be an attribute set with one attribute that is required.
And in case you were curious, creating a function that actually
takes two arguments is done the same way it's done in Haskell,
which I'm not going to get into here (it would look a little like this:
x: y: x + y
).
.
Specifically, we require one argument named pkgs
,
though ...
allows us to have additional arguments.
Just to see what would happen, I took the ...
out and ran my config to be greeted with
error: function 'anonymous lambda' called with unexpected argument 'config'
.
Also, notice how the braces that surround the function implementation do not
require a semicolon at the end, whereas the ones following an assignment do.
And since we've talking about functions,
that last code block also has two function calls in it.
The first, builtins.fetchTarball "github.com/some_url"
is us calling
the fetchTarball
function that happens to be located in the
builtins
attribute set.
This function takes a string as its only parameter
and returns a string representing the location of the downloaded files.
The second, (import "${home-manager}/nixos")
is us calling the import
function with an expanded string of
Home Manager's location as the lone argument.
It returns the nix expression (loaded and parsed) from the given location.
I've spent a whole lot of time getting the server to a state where I can happily work on it. It's been about 4 days since I even started writing this blog post, and at this time I haven't even begun to revise my drafts. I want to reiterate though, the next time I set up a new server, I can copy and paste my configs (honestly, I'll probably copy and paste directly from this blog post) and be done with it in 20 minutes.
Bringing my Services Back
I'm nearly back to the beginning! All I need to do now before I can create the blog is get my services running!
SSH
I know ssh
has already been enabled by line 15 of my main config,
but I'm removing that in favor of a slightly more thorough approach:
more info here
nix
/etc/nixos/configuration.nix
The newly added lines...
services.openssh = {
# Since I just removed line 15 which enabled SSH
# before, I now have to make sure I enable it here.
enable = true;
# Just for extra security, only allow logins using keys.
settings.PasswordAuthentication = false;
# Honestly, I just copied this line from the wiki...
settings.KbdInteractiveAuthentication = false;
# No root login!
settings.PermitRootLogin = "no";
};
Nginx
First, I want to add in a base level config for Nginx. So far, it is mostly just recommended settings copied from here. I'll be adding the virtual hosts soon.
nix
/etc/nixos/configuration.nix
The newly added lines...
services.nginx = {
enable = true;
# Use recommended settings
recommendedGzipSettings = true;
recommendedOptimisation = true;
recommendedProxySettings = true;
recommendedTlsSettings = true;
virtualHosts."TODO_MAKE_VIRTUAL_HOST_1" = { };
virtualHosts."TODO_MAKE_VIRTUAL_HOST_2" = { };
};
Also, to be able to use HTTPS easily, I add this acme config. This will be referenced later.
nix
/etc/nixos/configuration.nix
The newly added lines...
security.acme = {
acceptTerms = true;
defaults.email = "timpbh@gmail.com";
};
Now to fill out the virtual hosts.
The first one I want to add is for a staging server I was hosting while working on a project.
The config for this webserver from inside my /etc/nginx/conf.d/nodesite.conf
file (on the old server) looks like so:
nginx
/etc/nginx/conf.d/nodesite.conf
Partial config from old server
server {
server_name petproject_staging.donvi.biz;
location / {
proxy_pass http://localhost:8080/;
proxy_set_header X-Remote-Addr $remote_addr;
}
location /api/ {
proxy_pass http://localhost:8080/api/;
access_log /var/log/nginx/petproject_staging.access.log;
access_log /var/log/nginx/petproject_staging.access.log postdata;
}
access_log /var/log/nginx/petproject_staging.access.log;
listen 443 ssl; # managed by Certbot
ssl_certificate /etc/letsencrypt/live/donvi.biz/fullchain.pem; # managed by Certbot
ssl_certificate_key /etc/letsencrypt/live/donvi.biz/privkey.pem; # managed by Certbot
include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
}
Translating this to a nix config is surprisingly easy (though I did need to
look up some of the settings).
For the most part, the settings either are already directly configurable through nix,
or I can use extraConfig
to directly append to the end of the section.
nix
/etc/nixos/configuration.nix
The newly added lines...
virtualHosts."petproject_staging.donvi.biz" = {
forceSSL = true; # Always use HTTPS
enableACME = true; # Mentioned before, we will get our certs through ACME.
locations."/" = { #
proxyPass = "http://localhost:8080/";
extraConfig = ''
access_log /var/log/nginx/petproject_staging.access.log;
proxy_set_header X-Remote-Addr $remote_addr;
'';
};
locations."/api/" = {
proxyPass = "http://localhost:8080/api/";
extraConfig = ''
access_log /var/log/nginx/petproject_staging.access.log;
'';
};
};
You'll notice that I omitted every line that had # managed by Certbot
on the end,
since those are now managed using acme
.
Also, I did not include the postdata
log format since that caused me a few
problems (if you didn't notice that, ignore it)...
Regardless, after saving configuration.nix
and running sudo nixos-rebuild
switch
I now have my new Nginx server serving that one pet project of mine!
Well, sorta. Nginx of course just forwards requests,
but that pet project won't be running until I set it up using systemd
.
Everything Else — An Escape*
As you've seen from the last section,
some services can be installed just by adding them to the system configuration.
For example, I never went out of my way to install Nginx,
all I did was add services.nginx = { enable = true; ... }
to my configuration.
While that has been super convenient and is how I plan to install any other software that has
been packed for Nix, not all software already is.
For example, the webserver I was just configuring Nginx for that I've been referring to as
petproject_staging
is a web server I wrote myself
(in Java, which will be relevant), and since I didn't package it for Nix,
it just isn't packaged for Nix.
While I could keep going and learn how to package software for Nix myself,
this honestly feels like a good time to escape.
Going into this migration, I figured it would be impossible to have everything
I need be defined cleanly in a single config file on a single server.
For one, a .jar
file can't be placed in a config.
It must live elsewhere.
Oh, and since part of the reason I run my own server is to rely as little as I can on other
infrastructure,
I'm not going to be hosting my private server's code on GitHub.
So then, I figure there will be some necessary mutable state
— some files that are required to be placed manually for my server to function —
on my server, and I'm okay with that.
As long as I can have a clean divide from the 'pure' world of Nix to my own messy world,
I can keep that state as small as possible and still know what's going on.
As far as what this will include: I have two jar
s that run web servers,
one pre-compiledSpoiler:
Turns out this doesn't work for everything
and I just got lucky trying to do it with a Java program first.
I will end up getting the Java website working in this section,
but the Rust API won't work the same way. Rust binary,
and some associated data that needs to be loaded into Postgres.
Seeing as I only have 4 things outside my Nix config to keep track of, I find this to be a good compromise. Now: on to running the Java server.
Running the Java Web Application
Again, keep in mind that here I am leaving confines of the pure Nix setup. Despite that though, its actually incredibly easy to get this running.
There are two main additions I need to make to the configuration.
The first is to create the user, which I am giving the boring name of
petproject_staging
.
The user will have the necessary dependencies installed, and a matching group to be a member of.
The second is the Systemd
service.
This will simply declare the service name, the user that runs it, the command to run, and a few
other
things.
Everything that I need to add to my config is as follows:
nix
/etc/nixos/configuration.nix
The newly added lines...
# The user account for my `petproject_staging` service.
users.users.petproject_staging = { # Declare / create the user
isNormalUser = true; # Ideally this would be false, but `createHome` only works when it is true
createHome = true; # Creates a home directory for the user. This is where I plan on placing the jar file.
group = "petproject_staging"; # The user's group. Must be a valid group (which is defined later)
packages = with pkgs; [ # To install the packages...
jdk17 # JRE, JRK, close enough
pandoc # The website happens to depend on this
];
};
users.groups.petproject_staging = {}; # Create the group and nothing else (for use above)
# This creates a system service that will start up my website on boot
systemd.services.petproject_staging = {
enable = true;
description = "Does whatever petproject_staging does";
unitConfig = { };
serviceConfig = {
User = "petproject_staging";
Type = "simple";
# For whatever reason I decided this is where I'd place the jar.
# It makes it easy when its in the user's default path.
WorkingDirectory = "/home/petproject_staging";
# Notice the location of Java, each user has it's own directory
# for software. Behind the scenes, its actually symlinks.
ExecStart = "/etc/profiles/per-user/petproject_staging/bin/java -jar live_demo.jar --server.port=8080";
Restart = "on-abnormal";
};
wantedBy = [ "multi-user.target" ];
};
And once again I run sudo nixos-rebuild switch
,
all the changes are applied and my website starts running.
Of Course It's Not That Easy
Towards the beginning of this post, I mentioned I had once before set up a server with NixOS. For that server, the only software I installed was Nginx, Postgres, and a Java web server I wrote. It had never occurred to me that I got lucky by choosing to run a Java application. This time, when I got to the point of installing my Rust application, I was met with an unfortunate error:
Could not start dynamically linked executable: /home/pinger/dvs_pinger
NixOS cannot run dynamically linked executables intended for generic
linux environments out of the box. For more information, see:
https://nix.dev/permalink/stub-ld
Right. And that link, https://nix.dev/permalink/stub-ld is actually a redirect to the first question on the FAQ about NixOS:
How to run non-nix executables?
NixOS cannot run dynamically linked executables intended for generic Linux environments out of the box. This is because, by design, it does not have a global library path, nor does it follow the Filesystem Hierarchy Standard (FHS).
There are a few ways to resolve this mismatch in environment expectations: [...]
So, this is interesting.
I was aware of this before, I just forgot about the implications.
I mean, I knew that configuration files weren't where they normally would be because
they were generated by Nix, and I knew that installed programs weren't where they would
normally
be.
The location I used to find the Java executable in the last section was
/etc/profiles/per-user/petproject_staging/bin/java
which isn't a normal Linux path,
and even that is just a symlink to
/nix/store/ynkz2gd0hkm60vd96wwg7cgifv1x22f4-openjdk-17.0.13+11/bin/java
which
absolutely isn't a normal Linux path.
Of course a compiled binary is going to have a similar struggle.
Something I find interesting with all of this is how any program (or script)
that solely relies on software installable through Nix doesn't need really any special work
(that's how I got lucky before).
Anyway, so much for trying to escape... Time to go through the FAQ and decide what to do. It looks like there are three main categories of options:
- Build/compile the rust binary on NixOS using Nix.
- Modify a prebuild binary to 'hard code' the different library paths directly in.
- Somehow fake the environment that the binary runs in to look like it normally would.
I'll be going with the first option. Numbers 2 & 3 both sound kind of hacky, and while that isn't bad on its own, everything I've read makes it sound like I'm going to end up investing time learning whichever option I choose, so I may as well go with the cleanest looking option: 1.
Building a Rust Binary on NixOS
So it is obvious that I need to build my binary from source,
but just building it wouldn't be all that Nix-like.
Again, everything (well aside from the source code itself I guess)
should be declared in a configuration file,
and from dealing with other software I know that the resulting binary will live in some weird
location that looks vaguely like /nix/store/xyz[...]xyz-dvs_pinger
.
But how do I do this?
Following the trail started by the error message brings me here: a page on Packaging existing software with Nix. So a package it is!
Since good writing about creating a package with Nix already exists, I'm instead going to devote this next bit to explaining just enough to get by.
So first: a bit of terminology.
I've been using the word package
thus far as a sort of catch-all for all the stuff
(binary, other files, etc.) needed to properly install a program.
As best as I can tell, this is the way the word is used in Nix documentation,
though there is another term that keeps popping up: a Derivation.
Since Nix is about reproducibility, package are built with Nix,
rather than for it.
A derivation then, is what we need to create for Nix to know what to do.
These derivations provide all the information required to build a package from scratch,
and all derivations are made (directly or indirectly) through the built-in
derivation
function.
A simple derivation may look like the following:
nix
sampleDerivation.nix
Sample derivation structure
derivation {
name = "some_name";
builder = "some_builder";
system = "some_system":
}
Where the name
is just the name of the package,
the builder
is some executable that will do the building,
and the system
is the system architecture to build for.
More commonly though, the mkDerivation
function is used.
Here is a sample similar to that on
this tutorial,
which again, goes more in depth than I plan on going.
nix
sampleDerivation2.nix
Sample derivation with mkDerivation
. Note: this is incomplete
# ignore the fact that pkgs isn't supplied, this is just a mini example
pkgs.stdenv.mkDerivation rec {
pname = "some_package";
version = "0.1.0";
src = pkgs.fetchgit {
url = "https://github.com/location_of_package";
rev = "65843584352465464acb654d646a146n654a6553";
sha256 = "sha256-bkjwAJsdwblskdjnNBwpjenaGfgirACK2JlN9pOqBWA=";
};
buildInputs = [...] # To keep this brief, I'm
configurePhase = "..." # omitting the supplied values.
buildPhase = "..." #
installPhase = "..." # Check the linked tutorial for more details
}
The benefits are already visible.
Here we: no longer have to worry about the system architecture;
can specify dependencies that are needed at build time;
can specify commands to be run at different phases of the build;
download the source code directly using git
(taking care of course to make sure the hash's match);
and it only gets better from here —
since we don't even need to use this one directly
What seems to be the preferred method to build a rust binary on NixOS is the
rustPlatform.buildRustPackage
function.
Quite a few languages have
language frameworks on Nix,
and this includes
the one I'm using for rust.
After reading through that, this is what I ended up with:
nix
/etc/nixos/dvs_pinger.nix
Newly created derivation for my rust program. Entire file.
# Reminder: the `{something}:` notation is used to specify the arguments of this nix expression.
# Here, there is only one argument that we take: `pkgs`.
# The `?` and following expression supply a default value for the packages function in case our caller doesn't provide one.
{
pkgs ? import (fetchTarball { # The argument `pkgs` is required, and if not provided we will use `pkgs` as found here:
url = "https://github.com/NixOS/nixpkgs/archive/a2a2367d73a57054966acb562112648149b611c8.tar.gz";
sha256 = "sha256:1d016ndkn0vnx2bkh4fy1b3q6n0h5j2dnvpvhpmkz6j9ddgw3k9c";
}) {},
}:
# Our function is a wrapper around this, and returns the resulting derivation.
pkgs.rustPlatform.buildRustPackage {
pname = "dvs_pinger"; # Need to give it a name
version = "0.1.0"; # and a version
# We need to fetch the source. In the name of being as self contained as possible (especially since this is a private
# piece of software) I'm hosting the git repository on this server. Bit messy, I know.
src = fetchGit { # Retrieve the source from some git repository
url = "file:///repos/dvs_http_api/"; # That git repository happens to be local.
rev = "7d9338bbad756230a35f4f6cec80dd364c69aa41"; # The specific commit we want to checkout before building.
};
# The has computed over all crates this package downloaded as dependencies. To find this value,
# write in a fake hash and run the derivation. It will complain and tell you the proper hash.
cargoHash = "sha256-px3eJcGSsZAuxBHx5cDVkfjVENdLQGa7ikMNfFxC9Js=";
}
Now with the derivation created, we need to actually make use of it. Luckily, that will all be familiar. It should actually be nearly identical to how I ran the Java website, just with one small modification.
nix
/etc/nixos/configuration.nix
The newly added lines...
users.users.dvs_pinger = { # As always, create a new user just for this
isSystemUser = true; # This user exists to run a service, not for a human.
group = "dvs_pinger"; # As always, give it its own group
# This is where things are a little different...
packages = [ # The only package we need here is the one we created the derivation for.
(pkgs.callPackage ./dvs_pinger.nix {}) # Here we use the package defined by our previously created derivation.
];
};
users.groups.dvs_pinger = {}; # As always, create a group just for this.
# Actually, since so many of the comments are the same as before, I'll leave most of them out.
systemd.services.dvs_pinger = {
enable = true;
description = "Runs the DVS Pinger service which keeps stats of the uptimes of services";
unitConfig = { };
serviceConfig = {
User = "dvs_pinger";
Type = "simple";
# Notice how the location of the binary is in a similar place to where Java was for the last service
ExecStart = "/etc/profiles/per-user/dvs_pinger/bin/dvs_http_api";
Restart = "on-abnormal";
};
wantedBy = [ "multi-user.target" ];
};
# Since the service only runs the application and nothing else, I need
# to configure nginx to map the specific api endpoint to this service.
services.nginx.virtualHosts."api.donvi.biz".locations."/secretEndpoint" = {
proxyPass = "http://127.0.0.1:8000/secretEndpoint/";
};
And with that, I now have my rust API configured to be build and run with Nix.
All I need to do to push an update is push my source code to the server,
update the hash in dvs_pinger.nix
,
and run sudo nixos-rebuild switch
.
Reflection
So, do I feel like this was a success? That this was worth it? Absolutely.
In my mind, this is how servers should be managed.
I do feel that people will look back on server management and say
Wow, I can't believe people used to just hand install programs —
and they didn't even use version control for their systems!
.
Or, maybe I've been the one missing something.
There probably is another way to manage exactly what's on a server in a very controlled way,
but I've never found it.
Also, based on previous interactions with Nix, along with reading I've found online, I found this particular migration really quite easy — almost too easy. Yes, I had that one problem while configuring Nginx, and I did have to learn more than I wanted to get my rust binary working, but other than that it's been a surprisingly smooth process. It really feels as though Nix is simple to work with, and it's only the poor learning process that makes it difficult.
Results
In the first section, I mentioned that I didn't even know what was running on my old server. Now, I know exactly what I have. And I don't just mean that I remember what is there — I have a single directory that holds all my main configuration files (of which only 3 need to be read), and within those files there are only 327 lines (including comments and blank lines) that fully explain everything my server does. The fact that this just is a thing now will forever amaze me.
Yes, my configuration files may be a little disorganized and difficult to read,
and I do have a single exception (a folder named /repos/
that contains the source for my Rust program),
but when I inevitably take a year-long break from working on my server again,
I'll be able to refresh myself with everything in under 20 minutes.
Remembering About the Blog
I started this post by laying out my intent to write a blog, though I never exactly got around to setting that up. For the most part, all of my writing has been into a Markdown document. Turning that into a web-facing blog will have to be another post...